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1. INTRODUCTION 

In the implementation of education system that focuses on achieving the specified outcomes, the 
quality of a programme has to be monitored continuously. Assessment of programme performance is crucial 
in the monitoring process to identify the required improvement and to provide solutions for several issues [1- 
3]. The solutions can be designed once the analysis of root cause for the issues is determined from 
brainstorming and research. 

There are various ways of evaluating the achievement of a programme. Since Outcome Based 
Education (OBE) was introduced, the programme outcome measurement was used and a set of performance 
criteria was established to decide whether the programme has achieved its goal. Direct measurements, 
indirect measurements or both ways were employed in every course to produce the score for each programme 
outcome [4-5]. 

In the programme assessment, measurement of programme outcomes (PO) was carried out from 
various courses and each course contributes data from various activities which results in large data to be 
processed. Moreover, large amount of man power is required and it takes a long time to achieve the results, 
which will increase the operating cost if the measurement is carried out offline and manually. The 
programme assessment can be done efficiently using a proper technique. It is believed that a sampling based 
approach of measuring the PO with the online system as the mechanism, may help to accelerate the 
programme assessment process. 
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Numerous reports have described the outcomes of the course and programme assessments. Most 
report focused on the types of assessments and techniques used in measuring the course outcomes and 
programme outcomes at the course level [7, 8]. There is no report detailed out the mathematical model used 
in the determination of the overall PO at the programme level. Some reports revealed that averaging of PO 
score was used in the final decision of the program attainment however, it is not clear that the types of 
averaging technique employed in the work [9]. 

This paper describes the programme assessment process and the sampling based method used in 
evaluating the performance of Electrical Engineering Degree Programme. In this approach, only selected 
number of students and courses were considered in the measurement. A mathematical model for determining 
the final PO scores was developed and employed in the programme assessment. The effectiveness of the 
method was examined by implementing it in assessing the PO of the Electrical Engineering Degree 
Programme. 


2. RESEARCH METHOD 

The programme assessment process used in the faculty includes a few stages such as performance 
indicator design, performance assessment matrix preparation, data entry, PO measurement and analysis, 
Completion of CQI report, PO attainment analysis & Reporting, Approval of the Proposed Actions, 
Monitoring and Improvement as shown in Figure 1. All level of academic staff starting from lecturer to the 
faculty management took part in the assessment process. 

At the faculty level, the performance indicator for each programme outcome was designed and 
discussed in the workshop before the programme is assessed. There are twelve POs for each programme as 
listed in Table 1. The performance assessment matrix was prepared and discussed at the Centre for Studies 
level. It describes the selected courses for each PO measurement, method of assessment, name of the person 
in charge and others [10]. 
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Figure 1. The Programme Assessment Process. 


The PO measurement was carried out by the course coordinator whereas the PO attainment analysis 
was performed by the programme coordinator. A web-based system was developed to assist the academic 
staff in measuring the PO and examining the programme performance. The system uses a mathematical 
model to calculate the PO scores automatically and efficiently. The process of developing the mathematical 
model is shown in Figure 2. Before the mathematical model was designed, the technique employed in the 
creation of the model was first discussed with the external examiner and statistician. Two techniques; the PO 
averaging and the De Morgan Probability theorem were examined by applying them on a few samples and 
the results were analysed. The optimum technique was employed in the model development. The model was 
then implemented in the programme assessment and its feasibility was examined. 
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Figure 2. Process of Developing Mathematical Model for Programme Outcomes Measurement. 
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Table 1. Keyword of the Twelve POs 

PO Keyword 
PO! Engineering Knowledge 
PO2 Problem Analysis 
PO3 Design Solutions 
PO4 Investigation 
POS Techniques, Resources & Tools 
PO6 Societal, Health & Safety Issues 
PO7 ~— Environment & Sustainability 
PO8 Ethics 
PO9 Communication 
POLO = Individual & Team Work in 

Multidisciplinary Setting 
PO11 Lifelong Learning 
PO12 Project Management & Finance 











It is important to select the suitable courses for the programme assessment before the PO 
measurement is carried out. The criteria used in the selection of courses were the core courses that directly 
addressed the programme outcomes and the minimum number of courses selected for each PO was three. In 
the PO measurement for an individual course, the PO score was measured from a few assessment 
components for example, the examination question, report and assignment that address the selected PO. In 
this sampling based approach, only 30% of the overall students were selected in the measurement if the total 
number of the students is 100 and above, and from this percentage, the proportion of good, moderate and 
weak students was 0.33. The PO for an individual course, PO; was calculated using Equation (1). 


N 
1 
PO,=—Y P, —m=1,2,..N 1 


where N is the total number of assessments used in the measurement and P is the percentage of the average 
mark for each assessment, which is given by Equation (2). 


1 < 
P=|—— )_M, |x100% J =1,2,...K (2) 
KM ; “ 
j=l 
where k is total the number of student involved in the measurement, M;is a mark scored by each student and 
Mris the full mark for the assessment. Table 2 shows example of PO score calculation for an individual 
course using the above equations. 


Table 2. Example of selected PO Measurement for an Individual Course. 








Course Name Course | 
Selected Programme POL POS 
Outcome 
Selected Course Outcome Col CO3 
Fae Assignment | Eom Assignment 2 
No Subjects Question 1 Question 3 
Full mark M; Full Mark Full mark Full mark 
= 20 M;= 100 Mr= 20 M;= 100 

1 Student 1 17 80 18 95 

2 Student 2 20 86 10 98 

3 Student 3 19 70 15 85 

4 Student 4 18 75 12 80 

5 Student 5 15 95 17 60 

6 Student 6 16 88 16 75 

7 Student 7 10 96 17 78 

8 Student 8 17 78 16 68 

9 Student 9 9 60 9 56 

10 Student 10 12 65 8 54 
Total 153 793 138 749 
Percentage; P 76.5 % 79.3 % 69 % 74.9 % 
PO score POI = 77.8% POS =71.97% 
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The overall PO score for multiple courses was computed using mathematical model which was 
derived from De Morgan Probability Theorem for mutually exclusive events as shown in (3). The De 
Morgan Probability Theorem has been used in researches, for example to proof the circuit lower bound for 
automatic learning algorithm and for detecting and diagnosing faults found in heavy-duty diesel engines [11, 
12]. In probability, events that are not associated together or occur separately are said to be mutually 
exclusive [13]. The rationale of using De Morgan Probability theorem is that the PO score for each course is 
measured from different students, activities, types of assessment and methods of assessment. Therefore, the 
PO score from multiple courses cannot be averaged. The probability of achieving the targeted level of 
programme outcomes, PO,, is given by 


POm = 1 — (P'(0) x P'(A)x...x P'(k — 1) PO,, =1-|P'(1)x P'(2)x...x P(e) (3) 


where m is PO’s number, k is the number of courses selected in the measurement, P(1) is percentage of 
course | not achieving the KPI, P’(2) is the percentage of course 2 is not achieving the KPI, Pk) is the 
percentage of course k of not achieving the KPI. 

Rewriting the equation using PO term and the probability theorem PO_C,; = 1 - P(k) which is the 
percentage of achieving the KPI is equal to one minus the percentage of not achieving the KPI (Blitzstein et 
al. 2015), yields (4), a more detail elaboration of the overall PO calculation. 


PO,, =1 (: POS }( PO=© )s hs {1-75&) (4) 
100 100 100 


where PO_C; is the PO score for course 1, PO_C> is the PO score for course 2 and PO_C; is the PO score for 
course k. For example, referring to Table 3, if there are four courses were selected in the measurement of 
PO1 and the PO score for course 1; PO_C, is 80%, for course 2; PO_C> is 70%, for course 3; PO_C3 is 60%, 
and for course 4; PO_C4 is 50%, then using (4) the overall PO1 for the programme is 83.2%. 





Table 3. Overall PO Score Calculated from PO of Selected Courses 








Course PO Score 
Name POI PO2 PO3 PO4 PO, 
Course 1 80% - 75% - PO,_C; 
Course 2 70% - 719% - PO,_C, 
Course 3 60% - - 715% PO,_C3 
Course 4 50% - 100% - PO,_C4 
Course 5 - 92% - 90% PO,_Cs 
Course 6 - 88% - 80% PO,_Ce 
Course 7 - 90% - 82% PO,_C7 
Course k - 60% 90% - PO,_C, 
Overall PO 83.20% =: 99.96% 100% 99.5% POn 


score 





The mathematical model was first implemented in 2012 where the performance of six programmes was 
evaluated based on the programme outcome. Before the mathematical model was used, it was tested on a few 
selected courses in a workshop by the Head of Programmes and Outcome Based Education (OBE) committee 
to ensure error free and correct calculation. Once it was confirmed that the mathematical model and the web- 
based system were functioning well, a workshop was conducted to train the course coordinators to use the 
web-based system for measuring the PO and for completing the CQI report which was generated 
automatically. 

The PO attainment analysis was presented to the faculty management and discussed in the Closing 
de Loop (CDL) workshop every semester. Actions to be taken to improve the programme outcomes and to 
solve issues related to curriculum, teaching delivery and the implementation of OBE were identified in the 
pre-CDL workshop. These proposed actions were further discussed in the CDL workshop for approval. 
Monitoring of the actions taken was carried out by the management in the department and faculty levels. 
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3. RESULTS AND DISCUSSION 

Figure 3 shows one of the PO score obtained from the selected courses. Even though there is 
variation in the PO score, using the mathematical model, the overall PO can be determined. Figure 4 shows 
the PO attainment of one of the programmes which was obtained using the mathematical model and 
sampling-based approach. Each PO has exceeded the key performance indicator (KPI) target (a score greater 
than 65%) which shows the programme is successful. 

The percentage of student population in achieving the PO score of greater than 65% in one of the 
programmes is shown in Figure 5. More than 98% of the students have achieved the required knowledge and 
skills. These results demonstrate that the sampling based approach enables the achievement of the 
programme to be evaluated and the effectiveness of the OBE implementation in the programme 
to be examined. 

The sampling based approach provides several benefits. With a small number of samples used in the 
work, the performance of the programme can be analysed and the percentage of the student population that 
meets the targeted KPI could be determined in the specified time frame. Sampling based approach has been 
used widely in several applications to provide clear picture on the performance and quality of the instruments 
and services. Blanco et al. [14] has used sampling based approach to identify quality of service for web 
service orchestration. Began et al [15] has employed sampling based approach to determine the planned 
starting time of each appointment. 
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Figure 3. A Sample of PO Score Contributed from the Selected Courses. 
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Figure 4. PO Attainment of the Programme Measured Using Sampling based Method 
and the Mathematical Model Embedded in the On-line System. 
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Figure 5. The Percentage of Student Population in Achieving 
PO Scores above the Targeted Value. 


4. CONCLUSION 

The programme assessment process and the measurement of PO using the sampling based approach 
have been described in this paper. A mathematical model, which was derived based on the probability 
theorem, has been developed and used in the web-based system to assist in the final decision of the PO score. 
The implementation of the sampling based approach in the assessment provides a good indication on the 
programme and the students’ performance. Using the criteria for the assessment, the sampling based 
approach demonstrates that the programme is successful and the students have gained the required 
knowledge and skills. 
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